(1) Field of the Invention
This invention relates to a system and method for recognizing and tracking objects in a digital video image, and more particularly to a system and method for automatically acquiring, over a period of time, a matrix of digital reference images relevant to a particular location for use in performing match-moving integration, or combination, of video streams, or virtual graphic insertions, in live or recorded video depictions of that location.
(2) Description of Related Art
Image recognition based match-moving compositing of video streams, also known as virtual image insertion, is an established technology for combining imagery such as, but not limited to, computer generated animations, with live or recorded video sequences. The technology is widely used in both cinematography and in the production of live and recorded sports events. One particular, popular example of the technology in use is the virtual insertion of the first down line in broadcasts of football.
A problem inherent in the technology is that of providing good reference images for the image recognition and tracking software to use in positioning and orienting the virtual insertions. Changes in camera positions, angles and movement, as well as changes in lighting conditions that are not well reflected in the reference images being used by a video insertion system can result in even the most sophisticated searching and tracking algorithms performing poorly and even failing. This type of problem may be particularly evident in productions involving outdoor venues such as, but not limited to, daytime sporting events where lighting conditions may change rapidly and continually throughout the course of an event.
What is needed to solve this technical problem is a system and method for automatically and accurately providing a robust set, or matrix, or relevant reference images. Such a reference matrix of images would preferably be both an accurate map of the venue as currently seen by the cameras being used as well as having historical data that may be reverted to when conditions change back to some previous state. It would also be desirable to have a matrix of images that is able to automatically expand to include new landmarks as they appear or become available.
The relevant prior art includes:
U.S. Pat. No. 5,264,933 issued to Rosser, et al. on Nov. 23, 1993 entitled “Television displays having selected inserted indicia” that describes an apparatus and method of altering video images to enable the addition of images, message, slogans or indicia in such a manner that they appear to be part of the original image as displayed. The application of the apparatus and method is particularly adapted to be used as advertising during sporting events or other events whereby an operator selects a portion of the video image display such as a portion of a tennis court, recognizes the selected portion and inserts a logo or advertising message into that selected portion. The message is inserted into the selected portion of the court independent of how the scene is being panned or viewed by the television camera. The apparatus employs a pattern recognition algorithm to recognize the specific selected image portion and by means of the image processor substitute the logo into the first image to provide a display which appears as if the inserted image was actually painted on or otherwise permanently positioned on the court. Thus the inserted images are seamlessly and realistically incorporated into the original even as the original scene is moved, panned, magnified, zoomed or otherwise altered in size or perspective. The resulting composite image appears to the television viewer to be absolutely real and present at the scene.
U.S. Pat. No. 8,599,317 issued to Shamur et al. on Dec. 3, 2013 entitled “Scene recognition methods for virtual insertions” that describes a method of adding a virtual insertion to an image, according to an embodiment, includes extracting dynamic features from an input image, associating the dynamic extracted features with dynamic reference features in a reference feature database, generating a camera model based on the associations, mixing a virtual insertion into the input image based on the camera model, and outputting an image containing both the input image and the virtual insertion. According to another embodiment, a method of adding a virtual insertion to an image includes generating a biased camera model using a statistically selected subset of a plurality of non-fixed regions of the image, locating fixed reference objects in the image using the biased camera model as an entry point for a search, generating a corrected camera model using the fixed reference objects in the image, and adding a virtual insertion to the image using the corrected camera model.
U.S. Pat. No. 8,548,276 issued to Honma et al. on Oct. 1, 2013 entitled “Image search apparatus, image scanning apparatus, image search system, database creation method, and database creation program” that describes an image search apparatus capable of communicating with an image scanning apparatus includes: a common area extraction portion to compare a plurality of registration images stored in the storage portion and to extract a common area including an identical or similar image among a plurality of registration images; a search area determination portion to determine a non-common area excluding the common area, as a search area; a transmission request portion to request transmission of a document image if, as a result of comparison by the comparison portion, a registration image in which the reference image is identical or similar to the partial image does not exist in the plurality of registration images; and a registration portion to, in response to a request by the transmission request portion, store a document image received from the image scanning apparatus as a registration image into the storage portion.
U.S. Pat. No. 8,391,542 issued to Avrahami on Mar. 5, 2013 entitled “Method for estimating the pose of a PTZ camera” that describes an iterative method of estimating the pose of a moving PTZ camera. The first step is to use an image registration method on a reference image and a current image to calculate a matrix that estimates the motion of sets of points corresponding to the same object in both images. Information about the absolute camera pose, embedded in the matrix obtained in the first step, is used to simultaneously recalculate both the starting positions in the reference image and the motion estimate. The recalculated starting positions and motion estimate are used to determine the pose of the camera in the current image. The current image is taken as a new reference image, a new current image is selected and the process is repeated in order to determine the pose of the camera in the new current image. The entire process is repeated until the camera stops moving.
Various implementations are known in the art, but fail to address all of the problems solved by the invention described herein. Various embodiments of this invention are illustrated in the accompanying drawings and will be described in more detail herein below.